WE CLAIM: 



1. An interactive information processing system 
responsive to human operator movement of physical objects 
comprising : 

5 a discriminator for discriminating physical 

objects and generating an electronic physical object image 
signal representative of physical attributes of a physical 
ob j ect ; and 

a menu driven physical object attribute 
10 definition system for accommodating operator storage of 

physical attributes of a plurality of physical objects 
utilizing the electronic physical object image signal from 
the discriminator. 



2. A system according to claim 1, wherein a 
15 plurality of different types of physical objects are 

provided which correspond to physical objects normally used 
in handling paper files in an office, and wherein the 
attribute definition system includes means for 
accommodating operator indication of the type of physical 
20 object represented by the physical object image signal. 



3. A system according to claim 2, wherein said 
physical objects include objects representative of at least 
two of the following types of physical objects: 
(a) file of papers; 
25 (b) mail tray; 
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(c) fax machine; 

(d) computer keyboard ; 

(e) computer mouse ; 

(f) pen; 

(g) memo; 

(h) marker pen; 

(i) eraser; 

(j) operator's hands in an open release 
position; and 

(k) operator's hands in a closed condition. 

4, A system according to claim 2, wherein said types 
of physical objects include the operator's hands in open 
and closed condition, and the following other types of 
physical objects: 

(a) file of paper; 

(b) pen; 

(c) eraser; 

(d) mail tray; 

(e) fax machine; 

( f ) keyboard ; and 

(g) mouse. 

5 . An information processing system according to 
claim 1, comprising: 

a file storage unit containing electronically 
stored information to be processed, including information 
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from the physical attribute definition system reflective of 
the physical objects to be used with the system; 

a link storage unit containing information 
reflective of a physical object being concurrently 
5 discriminated by the discriminator; and 

a file operator for processing information from 
the file storage unit in response to operator movement of 
at least one of the physical objects with respect to an 
image display reflective of at least one other of the 
10 physical objects. 

6. An information processing system according to 
claim 2, comprising 

a file storage unit containing electronically 
stored information to be processed, including information 
15 from the physical attribute definition system reflective of 

the physical objects to be used with the system; 

a link storage unit containing information 
reflective of a physical object being concurrently 
.discriminated by the discriminator; and 
2 0 a file operator for processing information from 

the file storage unit in response to operator movement of 
at least one of the physical objects with respect to an 
image display reflective of at least one other of the 
physical objects. 

25 7. An information processing system according to 

claim 2, comprising a display surface in the form of a desk 
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top, said physical objects being selectively placed on the 
display surface by the operator. 

8 . An information processing system according to 
claim 7, wherein said discriminator is a camera which 

5 continuously views the display surface during information 

processing operations . 

9. An information processing system according to 
claim 7, wherein said attribution definition system 
includes an operator selectable menu at the desk top which 

10 menu includes operator selection areas for operator 

designation of a plurality of types of physical objects to 
be used by the operator, whereby the operator can 
sequentially place the physical objects to be used on the 
display surface and store their physical attributes, 

15 preliminarily to conducting actual information processing 

operations using the physical objects. 

10. An information processing system according to 
claim 5, wherein said file storage unit includes 
information reflective of predetermined distances between 

2 0 respective ones of the physical objects and image display 

reflection of the physical objects, and wherein said file 
operator includes means for initiating predetermined 
processing steps in response to movement of a physical 
object and an image display to positions within and without 

2 5 said predetermined distances. 
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11. An information processing system according to 
claim 10, wherein said information reflective of 
predetermined distances is indicative of characteristic 
point coordinates on said respective physical objects and 

5 image displays. 

12. An information processing system according to 
claim 11, wherein said characteristic point coordinates 
correspond to predetermined geometric locations on said 
physical objects and said image displays. 

10 13. An information processing system according to 

claim 11, wherein said characteristic point coordinates 
includes a point on a hand of the operator. 

14 . An information processing system according to 
claim 5, wherein said file operator includes hand gesture 

15 recognizing means for recognizing operator hand gestures 

reflecting release and hold operations for associated 
electronic files to be manipulated. 

15. An information processing system according to 
claim 14, wherein said hand gesture recognizing means 

20 recognizes a closed operator hand as a hold gesture and a 

hand gesture with open fingers as a release gesture. 



16. An information processing system according to 
claim 14 , wherein said hand gesture recognizing means 



recognizes the release and hold gestures by the size of the 
image reflective of the distance of the hand from a camera 
of the discriminator. 



17. An information processing apparatus comprising: 
5 a display unit; 

a camera unit for photographing an image object 
displayed on said display unit, and a physical object and 
the hands of an operator placed on said display unit; 

recognition means for recognizing the physical 
10 object from an image from said camera unit; 

attribute definition means for defining an 
associated motion between the physical object recognized by 
said recognition means and said image object; 

attribute association processing means for 
15 executing an associated motion defined by said attribute 

definition means for said image object when a comparison 
value between a characteristic point coordinate of the 
physical object recognized by said recognition means and 
another characteristic point coordinate of another physical 
2 0 object, or between characteristic point coordinates in said 

image object is within a predetermined value; and 

means for automatically starting said attribute 
association processing means when said image object or said 
physical object is recognized to be a hold-state by gesture 
25 recognition means. 
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18. An information processing apparatus provided with 
a display unit with a display surface on a horizontal 
arrangement in which data can be displayed in a document 
sheet format, and a camera unit for photographing the hands 
5 of an operator placed on the display surface of said 

display unit as well as the data displayed on said display 
surface of said display unit, comprising: 

definition file store means for storing various 
information processing motions each corresponding to each 
10 of various states of images representative of hand 

gestures ; and 

process execution means for executing a 
particular information processing operation for said data 
upon recognition that a distance between an image of the 
15 hand and said data is judged to be within a predetermined 

value according to images from said camera unit, and that 
an association for carrying out said particular information 
process operation intended by the operator is identified 
between the image of the hand and said definition file 



19 . The information processing apparatus according to 
claim 18, wherein a set of information processing 
operations corresponding to an image of a physical object 
which is physically corresponding to an image of a physical 
25 object which is physically placed on said display surface 

are stored in said definition file store means, and wherein 
said process execution means is caused to execute a 
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designated information processing operation corresponding 
to the physical object with respect to said data when said 
data is moved through manipulation of the image of the hand 
toward the physical object on the display surface as much 
5 as to come into a predetermined distance. 

20. The information processing apparatus according to 
claim 19, wherein an icon instead of said physical object 
displayed on said display unit is used. 

21. The information processing apparatus according to 
claim 19, wherein said data moved within said predetermined 
distance is highlighted. 

22 . The information processing apparatus according to 
claim 20, wherein said data moved within said predetermined 
distance is highlighted. 

15 23 . An information processing apparatus provided with 

a flat panel display capable of displaying a document 
object including plural pages each being partially 
overlapped, comprising: 

layout coordinate designation means for 
20 . designating a start point coordinate and an end point 
coordinate for layout ing said document object; 

page layout coordinate calculation means for 
obtaining respective coordinates for layouting respective 
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pages of said document from said start and said end point 
coordinates ; and 

layout display means for displaying respective 
pages of said document on respective coordinates obtained 
5 by said page layout coordinate calculation means. 



24. The information processing apparatus according to 
claim 23, wherein a layout coordinate point obtained by 
said page layout coordinate calculation means is an 
interpolated coordinate point which is obtained by 

10 interpolating a start point coordinate, a coordinate, and 

between said start point and said end point coordinates . 

25. The information processing apparatus according to 
claim 18, further comprising: means for simultaneously and 
dynamically displaying a curve which links between 

15 respective layout coordinates of the document object during 

layout operation by said layout coordinate designation 
means, and said document object as well. 

26. The information processing apparatus according to 
claim 23, wherein said layout coordinate designation means 

2 0 monitors and detects characteristic points of the both 

hands of the operator by means of a monitor camera thereby 
to designate respective coordinates. 

27. A method of information processing using an 
electronic computing system and a plurality of operator 
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manipulated physical objects corresponding to traditionally 
used objects for handling paper files in an office, 
comprising : 

selecting a plurality of physical objects; 
5 electronically imaging a first one of the 

physical objects, and generating a first electronic image 
signal reflective of the first physical object, utilizing 
an object attribute definition system menu to manually 
classify the first physical object by object type and to 
10 store the first electronic image signal with its object 

type; 

electronically imaging a second one of the 
physical objects and generating a second electronic image 
signal reflective of the second physical object, utilizing 

15 the object attribute definition system menu to manually 

classify the second physical object type and to store the 
second electronic image signal with its object type; and 

subsequently processing information in a file 
storage unit of the electronic computing system by moving 

2 0 at least one of the physical objects with respect to an 

image display reflection of at least one other of the 
physical objects. 

28. A method of information processing using an 
electronic computing system according to claim 27, wherein 
25 said physical objects include the operator's hands in open 

and closed conditions and at least one of the following 
types of physical objects: 



(a) 


file of papers ; 


(b) 


mail tray; 


(c) 


fax machine; 


(d) 


computer keyboard; 


(e) 


computer mouse ; 


(f) 


pen; 


(g) 


memo ; 


(h) 


marker pen; 


(i) 


eraser; and 



10 comprising electronically imaging and manually 

classifying each of the physical objects and storing their 
images along with their selected object type. 

29. An information processing system responsive to 
user movement of a physical object, comprising: 
15 a display unit, 

a physical object discriminator, 

a file storage unit containing electronically 
stored information, 

a link storage unit containing information 
20 reflective of a physical object discriminated by the 

discriminator and corresponding information in the file 
storage unit, 

and a file operator for processing information 
from the storage unit and link storage unit to accomplish 
25 interactive processing of information from the file storage 

unit in response to at least one characteristic of the 
physical object. 



30. A system according to Claim 29 , wherein said 
discriminator is operable to discriminate user hand 
movements - 

31. A system according to Claim 30, wherein said 
5 discriminator is operable to discriminate a plurality of 

different hand movements and to provide the link storage 
unit with a corresponding plurality of discrete signals 
which are processed by the operator to display respective 
different displays. 

10 32. A system according to Claim 29, wherein said 

discriminator includes a camera. 

33. A system according to Claim 29, wherein said file 
operator is operable to retrieve electronic files in 
response to placement of a physical object in a position to 

15 be discriminated by the discriminator. 

34. An information processing system using a user 
manipulated object, comprising: 

a display device, 

file storage means for storing electronic 
2 0 information, 

discrimination means for discriminating an object 
indicated by a user, 

link storage means for storing link information 
which designates the correspondence between said object and 
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the electronic information stored in said file storage 
means , and 

file operation means for reading the electronic 
information corresponding to the discrimination result of 
the object by said discrimination means from said file 
storage means on the basis of said link information and for 
displaying it on said display device. 

35. An information processing system using a user 
manipulated object, comprising: 
a display device, 

file storage means for storing electronic 
information, 

discrimination means for discriminating an object 
indicated by a user, 

link storage means for storing link information 
which designates the correspondence between said object and 
the electronic information stored in said file storage 
means , 

link definition means for generating link 
information for displaying the electronic information 
corresponding to the discrimination result of the object on 
said display device, 

means for displaying the discrimination result of 
the object by said discrimination means and the electronic 
information stored in said file storage means, and 



means for designating the correspondence between 
the selected electronic information among said displayed 
electronic information and said discrimination result . 



36. An information processing system according to 
5 Claim 35, wherein the discrimination result by said 

discrimination means includes at least information 
indicating whether the indicated object identification 
number and said object are indicated or not. 

37. An information processing system using an object 
10 according to Claim 3 5, wherein said link definition means 

operates when the link information for designating the 
correspondence between said indicated object discrimination 
result and electronic information is not stored in said 
link storage means . 

15 38. An information processing system using a user 

manipulated object , comprising: 
a display device, 

file storage means for storing electronic 
information, 

20 discrimination means for discriminating an object 

indicated by a user, 

link storage means for storing link information 
which designates the correspondence between said object and 
the electronic information stored in said file storage 

25 means, and 
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link definition means for displaying the 
electronic information corresponding to the discrimination 
result of the object on said display device and displaying 
a dialogue screen for accepting a change in said stored 
5 link information on said display device. 

39. An information processing system using an object 
wherein said system comprises: 

a display device, 

file storage means for storing electronic 
10 information, 

discrimination means for discriminating an object 
indicated by a user, 

link storage means for storing link information 
which designates the correspondence between said object and 
15 the electronic information stored in said file storage 

means, 

attribute discrimination means for discriminating 
the attribute information added to said object, and 

file operation means for executing the operation 
20 on the basis of said discriminated attribute information 

for the electronic information having the correspondence 
with said discriminated object. 

40. An information processing system using an object 
according to Claim 39, wherein the attribute information 

25 added to said object incudes at least information for 

designating whether the electronic information 
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corresponding to said object is allowed to be modified or 
not . 



